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As early as 2004, the Photovoltaic Thermal Control System (PVTCS) for the 
International Space Station’s 2B electrical power channel began slowly leaking ammonia 
overboard. Initially, the operations strategy was “feed the leak,” a strategy successfully put 
into action via Extra Vehicular Activity (EVA) during the STS-134 Space Shuttle mission. 
This recharge was to have allowed for continued power channel operation into 2014 or 2015, 
at which point another EVA would have been required. In mid-2012, the leak rate increased 
from 1.51bm/year to approximately 51bm/year. As a result, an EVA was planned and 
executed within a 5 week timeframe to drastically alter the architecture of the PVTCS via 
connection to an adjacent dormant thermal control system. This EVA, US EVA 20, was 
successfully executed on November 1, 2012 and left the 2B PVTCS in a configuration where 
the system was now being adequately cooled via a different radiator than what the system 
was designed to utilize. Data monitoring over the next several months showed that the 
isolated radiator had not been leaking, and the system itself continued to leak steadily until 
May 9th, 2013. It was on this day that the ISS crew noticed the visible presence of ammonia 
crystals escaping from the 2B channel’s truss segment, signifying a rapid acceleration of the 
leak from 51bm/year to 51bm/day. Within 48 hours of the crew noticing the leak, US EVA 21 
was in progress to replace the coolant pump - the only remaining replaceable leak source. 
This was successful, and telemetry monitoring has shown that indeed the coolant pump was 
the leak source and was thus isolated from the running 2B PVTCS. This paper will explore 
the management of the 2B PVTCS leak from the operations perspective. 


Nomenclature 

/? = Solar Beta Angle 

NH 3 - Anhydrous Ammonia 

DC = Direct Current 


I. Introduction: Space Station Electrical Power and the PVTCS 




T HE eight primary power channels of the International Space Station (ISS) Electrical Power System (EPS), 
corresponding to the eight solar array wings on the station’s truss, are the core of the United States Orbital 
Segment (USOS) power 
architecture. One of the 
primary purposes of the ISS 
integrated truss structure is to 
support these eight power 

channels, with two channels ^ sarj jSLi 

located on each of the four ^ so 
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modules, and each supports two power channels as shown in Figure 1. Each PVM contains the components needed 
to provide 150- 160V DC power to downstream systems and payloads, and batteries to allow uninterrupted power 
through the ISS’s orbital eclipse/insolation cycle. All USOS EPS and NH 3 cooling is the operational responsibility 
of the Station Power, ARticulation, Thermal, and ANalysis (SPARTAN) officer within NASA’s Mission Operations 
Directorate (MOD), located at the Mission Control Center in Houston (MCC-H). 

A. Photovoltaic Thermal Control System (PVTCS) 

As battery discharge is an exothermic reaction, active cooling is necessary to keep batteries below upper 
operating temperature limits. Cooling is also necessary for the various electrical components on the PVM involved 
in power routing and DC-to-DC voltage conversion. This active cooling is provided by the circulation of NH 3 in the 
Photovoltaic Thermal Control System (PVTCS), which uses a Pump Flow Control Subassembly (PFCS) orbit 
replaceable unit (ORU) to circulate coolant through the Integrated Equipment Assembly (IEA) onto which the EPS 
ORUs are attached. The PFCS contains two pumps for redundancy, with only one pump active at a time. The PFCS 
also contains a Flow Control Valve (FCV) for temperature control. The FCV rotates as needed to force NH 3 to 
either bypass or flow through the Photovoltaic Radiator (PVR) based on the battery temperatures. Though each 
channel’s PVTCS loop is independent, both loops on a given PVM flow through the same PVR ORU. At time of 
launch, each PVTCS contains approximately 531bm of NH 3 , of which approximately 131bm is surplus to provide for 
some amount of leakage. Figure 2 shows a schematic of the PVTCS. 

B. EPS Redundancy 

The electrical power system’s eight-channel architecture allows for great flexibility in terms of response to 
component malfunctions. Loss of a power channel can result from a variety of malfunctions occuring in a number of 
ORUs or in the wiring between them. In the case of an electrical short, channel loss may be instantaneous. In the 
case of PVTCS loss, channel loss will occur within hours as the channel batteries reach their maximum operating 
temperature. 



- ■ 

Figure 2. PVTCS Schematic. For the P6 PVM, the 2B PVTCS would be the upper loop as shown, with 4B below. 
Orbit Replaceable Units (ORUs) are denoted by dashed lines. 
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The failure of any single power channel can be allayed by handing over that channel’s end user loads to another 
channel via such EPS ORUs as the Main Bus Switching Unit (MBSU). For instance, in response to a malfunction in 
late 2012 of the 3 A channel, a relay was commanded closed within MBSU 3 to provide power to all of channel 3A’s 
loads from the 3B channel. This, combined with some parallel power architecture downstream of the MBSUs allows 
for the operational flexibility to sustain the loads of a lost power channel while not overtaxing any other channel. 

The core functions of the USOS are divided between three power channels: 1A, IB, and 2B. These core 
functions include the avionics and thermal control of the US Lab module - both the internal water cooling and the 
external NH 3 cooling. 2B is the only channel which powers both an internal thermal control loop as well as an 
external loop. 2B also powers one of two S-band communication systems which is the primary communication path 
between ISS and MCC-H for voice, commanding, and telemetry. Therefore, the 2B channel is arguably the most 
critical power channel on ISS. 

Loss of the 2B channel alone is not catastrophic; loads can be shifted, primarily to the 2 A channel. However, the 
presence of so much critical equipment on the 2B channel puts a large power demand on whatever channel or 
channels would end up supporting its loads. The aggregate effect of a long-term loss of the 2B channel is a loss of 
redundancy and a decreased capability to respond to any subsequent ISS failure. Furthermore, though the loads can 
be shifted, the overall supply of power to the USOS is reduced by approximately 12.5%*. These reductions are 
exaggerated whenever solar arrays are not autotracking the sun such as when they are positioned for vehicle 
dockings/undockings, thruster activity, etc. 

II. Identification of 2B Ammonia Leak 

Determining the trend of fluid mass in an ISS NH 3 system is not a straightforward task, as the system provides 
data from a limited set of temperature, pressure, and fluid quantity sensors. This, combined with the effects of 
orbital cycles, perturbations of FCV operation, and the -75° to +75° variance of the solar p angle, leaves the 
difficulty in trending a fluid leak inversely proportional to the size of any such leak. Nevertheless, by the middle of 
2008, Boeing thermal systems engineers noticed that the 2B channel’s PVTCS had been losing NH 3 mass at a rate of 
approximately 0.751bm/yr to 
1.51bm/yr since 2004 (see Figure 
3). If enough NH 3 were to leak 
out, a Fault Detection Isolation 
and Recovery (FDIR) software 
algorithm would automatically 
shut off the pump to prevent 
pump cavitation and damage. 

A leak of this size was far too 
small to be detected visually, as 
the orifice associated with such a 
massflow would be microscopic. 

Therefore, a “feed the leak” 
strategy was agreed to by the ISS 
Program, whereby spacewalks 
would occur every few years for 
the purpose of topping off the 
NH 3 reserve. By the middle of 
2009 the ISS Program 
determined that the first such 
refill would occur during the 
STS- 134 mission, also known as 
ISS flight ULF-6, which at the 
time was scheduled for mid-2010. This would allow the refill to be completed ahead of the 2B PVTCS reaching its 
automated FDIR shutdown point due to low quantity levels, which was estimated to occur in the middle of 2013. 



Figure 3. Expellable PVTCS 2B fluid mass (lbm), 2001-2012. Expellable mass 
of 0 represents the point at which the 2B PVTCS would need to be shut down to 
prevent pump cavitation. Mass is calculated based on a small set of temperature, 
pressure, and fluid quantity sensors located within the PVTCS, and can be biased 
by thermal environments, attitude, and whether or not the PVTCS is unpowered 
(as it was for a time in late 2007 when the P6 truss was relocated). The dotted 
line is the STS-134 PVTCS refill described in section III. Note rate increase in 
2012 as described in section IV. 


As the 2B solar array, along with 4B, has been on orbit the longest and has degraded with time, total power 
capability is slightly less than 12.5% of total USOS capability. 
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Figure 4. PVTCS Refill Pipeline schematic 


III. PVTCS Refill Concept and Execution 

A. Refill Architecture 

As the leak of a PVTCS was a known possibility from the early days of ISS design, the integrated truss structure 
was designed to facilitate the refilling of a PVTCS from the much larger NH 3 stores of the External Thermal Control 
System (ETCS) via a series of in-situ pipes and EVA-installed jumpers (see Figure 4). 

The ETCS is the NH 3 system responsible for the cooling of the habitable modules of the USOS, in addition to a 
number of critical EPS ORUs located on the truss itself. Compared to the eight PVTCS loops, each with ~551bm of 

NH 3 , there are only two ETCS loops, 
etcs Loop b each w ith approximately 5501bm of 
NH 3 plus a further 6601bm within each 
loop’s Ammonia Tank Assembly 
(ATA). The ATA is pressurized by the 
Nitrogen Tank Assembly (NT A), 
containing a commandable Gas 
Pressure Regulator Valve (GPRV), 
downstream of a solenoid N2 Supply 
Valve, which was itself downstream of 
the NTA’s nitrogen tank. There is no 
cross-connectability of one ETCS loop 
to another. As both ETCS loops are 
mounted to the central section of the 
truss in proximity to the habitable 
modules, there is normally no flow 
between either ETCS and any PVTCS, as the constantly-rotating Solar Alpha Rotary Joint (SARJ) is in between. 
The refill pipeline for the P6 PVTCS starts at EVA-configured fluid jumper panel A500. At this panel located on the 
PI truss, a small 0.25” diameter, 16” long fluid jumper can connect any two of the three ports: the supply from the 
mechanical ATA overboard vent valve, the path to an overboard vent, and the refill pipeline. 

Outboard of this plate, the in-situ tubing continues to a male quick disconnect (QD) at the outboard end of the PI 
truss segment. From here, another EVA-installed jumper connects to a similar QD on the inboard end of the P3 truss 
segment (there is no P2 truss segment). This connection was already in place at the start of STS- 134, as the 
connection of this and similar truss-to-truss jumpers had been completed on previous EVAs. A similar 
hardline/jumper configuration connects P4 to P5. 

Immediately inboard of the PVTCS is a section of P6 called the long spacer. The long spacer contains the 
unique architecture of the Early External Thermal Control System (EETCS), which was designed into the P6 truss to 
support US Lab cooling from 2000-2006 before the ETCS was assembled with the launch of the SO, SI, and PI 
trusses 1 . The EETCS fluid system is separate from P6’s two PVTCSs, and utilizes two of its own radiators, 
deployed orthogonally to the P6 PVTCS radiator. 

From its launch in 2000 until 2007, the P6 truss sat atop the ISS centerline versus its current position as the most 
outboard port segment of the truss. The EETCS was deactivated in 2007 when the ETCS was activated to provide 
heat rejection for the US Lab. In October 2007, with the port-side truss assembled out to P5, the P6 truss was 
robotically conveyed from its central position near the US Lab out to the end of P5 where it remains today. The 
EETCS also acts as one of the final segments of NH 3 travel during a P6 PVTCS refill, as some of the tubing of the 
dormant EETCS is utilized as an NH 3 passage. From P5, NH 3 flows through a fluid jumper into the P6 EETCS lines, 
and then through a final jumper into the P6 PVTCS known as the Early Ammonia Servicer (EAS) jumper (described 
in section IV). 

With much of the architecture in place by design or as a result of EVAs leading up to an eventual PVTCS refill 
(see Figure 4), the remaining tasks for the STS-134/ULF-6 crew to enable a refill were to: 


• Vent the nitrogen pad from the refill pipeline 

• Configure the A500 panel for the PVTCS fill (and return to normal at EVA completion) 

• Connect P3 to P4 via a jumper across the SARJ, which would be locked for the EVA 

• Connect P5 to the EETCS 

• Connect the EETCS to the 2B PVTCS via the EAS jumper 
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Once the desired connections were in place, MCC-H could commence the refill operation via ground- 
commanded valve operation. 

B. From Refill Design to Execution 

When the MOD team was notified in 2009 that the PVTCS 2B refill was being added to the STS- 134 task list, 
the assigned flight control teams immediately started examining the never-before-used refill architecture. The 
concept of the refill was straightforward: connect the large ETCS to the small PVTCS via a series of jumpers, and 
use regulated nitrogen pressure to command the proper amount of ammonia transfer. However, the MOD team 
quickly determined that the PVTCS refill would be an extremely dynamic operation. 

Over the 18 months before the STS- 134 mission, the MOD flight controllers responsible for the ETCS and 
PVTCS collaborated with thermal engineering teams, spacewalk officers, astronaut crewmembers, and safety & 
mission assurance personnel to develop the nominal and contingency procedures which would be utilized to execute 
the design concept of the PVTCS refill pipeline. The procedures would have to ensure a precise fill, as underfilling 
would require a refill sooner, and overfilling could potentially lead to overpressurization of the PVTCS in certain 
thermal environments. Also, it was critical that the fill be executed in such a way to mitigate the risk of the crew 
spacesuits being contaminated with the toxic NH 3 when ingressing the habitable modules at the end of the EVA. 
Furthermore, the risks to the onboard thermal systems themselves were evaluated, as a leak or malfunction of any 
component could have tremendous impact on future ISS operations. 

The most significant constraint from the thermal systems perspective was that while the ETCS was in fluid 
contact with the EETCS or PVTCS, at least one closed valve needed to separate the PVTCS from the nitrogen 
supply tank. This constraint was designed to mitigate the risk that a sudden failure of the GPRV, which normally 
keeps ETCS loop pressures above 2000kPa, would cause pressures to increase above the 1551kPa maximum 
operating pressure of the EETCS & PVTCS. This constraint drove most of the complexity of the refill procedure. As 
a result, it was decided that a known safe pressure would be set in the ATA at the start of the EVA, and this pressure 
would be used once the pipeline was assembled to push NH 3 into the PVTCS. If the end pressure was too much, it 
could simply be vented by the GPRV as long as the nitrogen supply tank was sealed off by closing the N2 Supply 
Valve - similar to the procedure performed when filling an automobile tire. If the pressure was too little, a 20- 
minute cycle of ground commands would be needed to isolate the ATA from the refill pipeline, repressurize the 
ATA from the NTA, isolate the nitrogen tank, and open the ATA to the refill pipeline once again. 

Another concern, which threatened to complicate the EVA timeline, was with hydraulic isolation of the pipeline. 
Following the PVTCS refill, the stretch of the refill pipeline from the PI truss to the P5/EETCS interface, as well as 
the jumper between the EETCS and PVTCS, would be vented. The lack of an accumulator to provide hydraulic 
compliance on either of these pipeline sections meant that venting was required for concern of overpressurizing and 
bursting the lines in the likely event that the truss was subjected to a warmer thermal environment. However, this 
venting also required that for some short period, the lines would have to be isolated before being vented. Normally, 
this would require that both pipeline sections be vented during orbital eclipse; however, placing such a requirement 
on an EVA timeline is undesirable as it is inefficient to pull a crewmember away from a task they are working on to 
go execute a separate, time-critical task at a different worksite. Therefore, MOD asked thermal engineers to perform 
analysis to determine, worst-case, how long it would take the isolated pipeline in direct sun to warm up above 
maximum certified pressure. Engineers deemed it acceptable to isolate the P1-P5 pipeline for 2.5 minutes even in 
insolation. 

C. The PVTCS Refill EVA 

EVA 2 of STS-134 took place on May 21 st of 2011. The ISS thermal system configuration at the start of the 
EVA was with the ETCS Loop B ATA isolated from the main circuit of the loop, and ATA Tank 2 pressurized to 
1551kPa in preparation for the fill. The PVTCS was operating nominally, with the exception that its FCV was 
commanded to hold a slightly cold-biased position rather than engage in closed-loop control. This was an effort to 
keep the PVTCS fluid dynamics in a steady-state configuration. 

Just after 0709 GMT, the refill operation began with the ground-commanded opening of the ATA vent valve to 
pressurize the pipeline from PI to P5. After confirming that ATA fluid quantities quickly stabilized and that the 
pipeline was not leaking, the EETCS was opened to the pipeline when EVA crew opened the P5/EETCS connection. 
Again, after watching the fluid quantities settle (this time with the added telemetry of the EETCS accumulator 
sensors) the jumper to the PVTCS was opened. Within minutes, PVTCS pressure increased from 900kPa at the 
pump inlet to over 1200kPa, and a 2B PVTCS fluid quantity sensor showed an increase of over 30%. After the 
thermal systems officer, per thermal engineering request, utilized the GPRV to slightly reduce the pipeline pressure 
(and, with it, the PVTCS pressure), the EVA crew was directed to disconnect the PVTCS from the pipeline. This 
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was performed at 0754GMT, less than an hour from the start of the fill. Later computations showed that a total of 
8.571bm of NH 3 was transferred into the PVTCS, just 0.161bm short of the targeted fill mass. The refill was a 
tremendous success, and the 2B PVTCS was returned to nominal operation by the end of the EVA. Assuming an 
unchanged leak rate, the system could operate for over 4 years before needing another refill. 


IV. 2012 Leak Rate Increase & US EVA 20 

A. EVA Necessity 

In September 2012, active thermal engineers determined from their long-term PVTCS 2B mass trending that its 
leak rate had increased, perhaps fourfold, and that the leak rate increase may be exponential. Due to the “noise” 
inherent in plotting the PVTCS mass this leak rate increase was only confirmed months after it began, and could not 
be precisely quantified (see Figure 3). Nevertheless, if the exponential trend were to continue, the 2B PVTCS would 
likely leak out by the start of 2013. 

EVA planning began almost immediately. Crew rotation and solar beta constraints made late October/early 
November 2012 the desired time for an EVA. The first decision to be made was whether to perform a PVTCS refill 
to try to sustain operations until a new crew arrived in 2013, or to attempt a troubleshooting EVA. Almost 
immediately, a refill was deemed to be inappropriate for a crew who had never rehearsed such an operation on the 
ground. Though it could certainly be done, the likelihood was that a crew without ground training would require 2 
EVAs, not 1, to perform the task. The decision was made to troubleshoot. 

Though it was possible to determine with some certainty where the leak was by isolating the three parts of the 
PVTCS - PFCS, PVR, and irreplaceable hardlines/coldplates - and monitoring them for leaks, the isolation would 
require that the 2B PVTCS remain offline for an extended period. This was deemed undesirable due to the criticality 
of the 2B channel loads. Another early idea was to simply replace the PFCS; however any gains in EVA simplicity 
would be overshadowed if the PVR was the leak source, a leak source that would have quickly depleted the system 
reserve. 

B. Proposed System Architecture Reconfiguration 

To allow the channel 2B PVTCS to keep running while attempting to isolate the leak source, ISS engineers and 
operations personnel crafted a plan to utilize a unique capability of the P6 truss segment. The 2B PVTCS would be 
cross-connected with the adjacent decommissioned EETCS via two EAS jumpers, and one of the EETCS radiators 
would be redeployed after 6 years of dormancy. In this configuration, the 2B flowpath within the P6 PVR could be 
isolated while the Trailing Thermal Control Radiator (TTCR) of the EETCS provided cooling to channel 2B. This 
configuration is shown in Figure 5. 

That this cross-connection capability exists at all is a testament to operational foresight. In the late 1990’s, the 
ISS Program asked MOD to develop some suggestions for hardware which could inexpensively be added to the ISS 
to mitigate potential failures instead of requisitioning additional expensive spare ORUs. The goal was to make the 
ISS a more resilient vehicle, better able to sustain operation if critical equipment failed, especially in the early stages 
of ISS construction. Until STS-1 15 launched the P4 PVM, the USOS only had two primary power channels: 2B and 
4B. If one of these channels were to fail, the difficulty of assembling the ISS would increase dramatically. One 
such potential failure was the leak of one of the two PVTCS loops. 

In order to help ensure electrical power system reliability in the early period of ISS assembly, the EAS was 
designed to be an EVA-operated pressurized NH 3 tank to refill a PVTCS after a leak was isolated. If such a leak 
was determined to be in one of the three radiators on the P6 truss - the PVR, the TTCR, or the Starboard Thermal 
Control Radiator (STCR) - that radiator would be isolated and the remainder of the loop would be refilled from the 
EAS. After the refill, two flexhoses (the EAS jumpers) would be put in place to carry the PVTCS NH 3 over to the 
EETCS Loop A radiators. This EAS implementation strategy was developed after construction of the P6 truss had 
already begun, and thus utilized quick disconnect ports already in place for fill-and-drain purposes. 

Though the EAS was never needed and was jettisoned from station in 2007 after the STS-1 15 mission was 
complete, the EAS jumpers remained. Their first use was in 201 1 when one was used as the final connection to the 
PVTCS during the 2B refill, and their full capability was realized in 2012 for EVA 21 when the EAS jumpers 
connected the ailing 2B PVTCS to the dormant EETCS. Though cross-connecting the PVTCS to flow through the 
EETCS was an established concept, the 5 weeks of preparation before EVA 20 were spent not just choreographing 
the spacewalk, but also involved a tremendous effort from thermal engineers to verify that the fluid dynamics of the 
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to-be-created “hybrid” system would allow for adequate cooling of the 2B EPS ORUs given the state of the ISS in 
2013 in terms of power demand, vehicle attitude, and other operational constraints. 



Figure 5. Schematic of PVTCS/EETCS “Hybrid Config” The 2B lines of the P6 PVR, shown top right, are 
isolated. Warm NH 3 leaves the 2B PVTCS out of QD-1, through one of the EAS jumpers, and enters EETCS Loop A 
via QD-10. From here, the NH 3 arrives at a T-junction. Half of the NH 3 flows through the deployed Trailing 
Thermal Control Radiator (TTCR, shown at top), while half flows through the retracted Starboard Thermal Control 
Radiator (STCR). Note that the radiator naming convention is based on P6 being in its original position atop the 
station centerline. The 2B PFCS, at right, contains the only running pump - the EETCS Loop A and B PFCSs are 
powered, but their pumps are off. All of EETCS Loop B is stagnant. 


C. US EVA 20 - November 1 st , 2012 

US EVA 20 was executed to put the above plan into action. First, a fluid connector valve was closed to isolate 
the channel 2B flowpath within the P6 PVR. Next, the EAS jumpers were configured so that PVTCS NH 3 which 
would normally flow to the PVR would be routed out through the jumper over to the EETCS with its to-be-deployed 
TTCR. Finally, the TTCR insulation shroud was removed and the TTCR was uncinched. Once the crew was clear of 
the radiator, SPARTAN successfully commanded deployment of the TTCR. The STCR remained retracted and 
covered by insulation; deployment was not an option due to interference with one of the solar arrays of the P4 truss 
(which was not an issue when P6 sat atop the vehicle centerline). 

V. Operating 2B PVTCS In the Post-EVA 20 “Hybrid Config” 

In the months following EVA 20, MOD and engineering personnel determined that the 2B PVR lines were not 
leaking and the combined PV+EE system was continuing to leak at approximately the same rate seen before the 
EVA. As of early May 2013, data trending indicated that due to the added fluid mass from the EETCS, channel 2B 


7 

American Institute of Aeronautics and Astronautics 



Figure 6. Overview of ISS port truss post EVA 20. 

The EETCS TTCR is shown upwardly deployed, while 
the P4 and P6 PVRs are deployed downwards. 


operation could continue until at least mid-2014 without EVA intervention. As far as day-to-day operations of the 
system, changes were few and minor, which meant retraining of ground operators was minimal. 

The largest operational concern was how to protect the PVTCS pump against cavitation resulting from a sudden 
NH 3 leak while not shutting down unnecessarily. Fault detection and isolation algorithms which monitored the 
accumulator quantities were smartly built with the ability 
for ground operators to force the software to ignore certain 
sensors - in this case, the sensors of the isolated radiator. 

However, the realities of this hybrid configuration were 
never assumed in the software design, and there was no 
automated FDIR leak algorithm which considered all of the 
relevant accumulators - the PFCS accumulator in the 
PVTCS, and the 3 accumulators in EETCS Loop A. If only 
the running pump’s accumulator was accounted for, the 
leak algorithm would be too conservative and shut the 
pump off before truly needing to. This was a risk as the 
fluid dynamics of the system resulted in that particular 
accumulator having the lowest fluid levels. In response, 

MOD took advantage of a capability to create and utilize 
simple automated algorithms within station software known 
as Timeliner 2 . A Timeliner algorithm was created to 
monitor all of the accumulators of the hybrid system, and 
would deactivate the pump when 5 of 6 quantity sensors 
(some accumulators have 1 quantity sensor, some have 2) all fell below a safe level. From this point, the procedural 
response would be the same as for the leak of any other PVTCS. 

System heat rejection capability was also altered with the change in 2B PVTCS architecture. The 2B PVTCS 
was made less capable to cool since NH 3 flow was split between the deployed TTCR and the retracted and non- 
cooling STCR. Furthermore, the TTCR is deployed in the opposite direction of the PVRs (see Figure 6), and, with 
SARJ autotracking, the TTCR’s long axis pointed into the sun. The end result was that the PVTCS 2B FCV spent 
more time, on average, in the full cooling position than other channel PVTCS FCVs. MOD worked with engineering 
pseronnel to establish a protocol for responding to a scenario where channel 2B fluid became too warm to protect 
against pump cavitation. In this case, the pump would be shut down when net positive suction pressure fell below a 
certain value. 

The 4B channel was affected by the 2B channel alterations, as channel 4B’s NH 3 had the P6 PVR all to itself and 
was therefore at risk of flowing colder NH 3 than the lower operations limits of certain sensors. The 
bypass/flowthrough algorithm of the PVTCS FCVs on all channels result in very cold slugs of NH 3 flowing through 
the system immediately following any FCV reposition to full cooling, and these slug temperatures fall very close to 
the lower FDIR limits. This problem was simply exacerbated on 4B to make an undertemp shutdown more likely in 
certain (3 and Port SARJ positioning combinations. In response, MOD worked with engineering personnel to 
establish a protocol to respond to a PVTCS undertemp by restarting the loop with the FCV at a fixed position. 
Meanwhile, engineering developed a matrix of (3 and Port SARJ positioning combinations which were most likely to 
trigger these events. 

For both 2B and 4B, these temperature exceedances would only occur if the Port SARJ is not autotracking the 
sun, or if the ISS were in a non-standard attitude. To date, neither the channel 2B overtemp or 4B overtemp 
protocols have ever been needed as all assumptions are based on worst-case thermal environments and FCV timing. 

There was also concern that the TTCR and P6 PVR each had only one active flowpath, and one stagnant 
flowpath. If the remaining active flowpath stopped flowing due to a failure, there could be freezing or overpressure 
concerns. The freezing concern of the P6 PVR was sufficient for engineering to request that at certain high betas, in 
the event of a 4B PVTCS failure, the Port SARJ be biased by 180° to point the P6 PVR’s long axis into the sun. In 
this case, power generation would not be affected as the solar array Beta Gimbal Assemblies (BGAs) would 
automatically flip themselves 180° to point towards the sun again. P4 PVR heat rejection would be reduced, too, by 
an acceptable amount. A similar protocol would need to be put into place any time a PVTCS loop was shut down for 
a long period, likely as the result of a loop failure. 

In summary, a number of contingency procedures were updated for PVTCS operation as new constraints were 
discovered with the hybrid configuration, so as to fulfill MOD’S goal of ensuring that an operator without intimate 
knowledge of the system’s history would still have the needed information to respond to off-nominal situations. 
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VI. May 2013 Visible Leak & US EVA 21 


A. Visible Leak and EVA Prep 

On May 9 th , 2013 (GMT 129) at approximately 10:30am CDT, ISS Commander Chris Hadfield called MCC-H, 
stating that the crew was seeing flakes emanating from the port truss. (Hadfield later stated that Pavel Vinogradov 
first saw the leak from the window of his sleeping quarters and notified the Commander of the “sparks” floating 
outside.) SPARTAN was also monitoring a decrease in 2B PVTCS quantity values at that time, but had initially 
assumed that the decrease was related to heat load reduction from a 2B battery capacity test in progress. 
Furthermore, the leak trend looked to be almost exactly the same type of trend commonly seen in the 2B post-EVA 
20 configuration during any SARJ parking event - though the SARJs were autotracking at this time (see Figure 7). 
The crew was immediately asked to point a high-definition video camera at the leaking truss segment, as the JEM 
port side window provided optimal viewing. 

Telemetry indicated a 51bm/day leak, with automated shutdown action expected to occur within 24-48 hours 
when the 2B PFCS quantities would fall below 4%. The Timeliner bundle discussed previously was not yet onboard 
and therefore EETCS quantities were not a part of the leak shutdown algorithm. 

Approximately 12 hours after the initial crew report of the leak, the relevant MBSU was configured to put all 2B 
loads onto the 2A channel. Once this was complete, the 2B PVTCS pump was commanded off. Within a number of 
hours, the visual appearance of flakes had stopped. 

At a 3:00pm Failure Investigation Team meeting, the ISS Program Manager (after coordination with MOD EVA 
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Figure 7. 2B PFCS Fluid Level (GMT 2013/124-129). Y- axis is the fluid quantity as measured by two sensors 
within the 2B PFCS accumulator. Each ‘beat’ is approximately an orbit. Within an orbit, quantity is highest when the 
FCV is in full-radiator flow, due to the effects of line resistance across the hybrid system. Note the similarity of the 
pattern of the GMT 129 leak to the two SARJ feathering events prior. 


personnel) stated that the goal of the team would be to perform an EVA on May 1 1th. The EVA crew would replace 
the channel 2B PFCS, as this is was the only possible remaining leak source in the 2B system having isolated the 
PVR on EVA 20. The only caveat was that a pump replacement would not be performed if it appeared to the crew 
that the leak source was obviously from the irreplaceable hardlines underneath the PFCS and EPS equipment. 

The desire for an unprecedently quick turnaround EVA was two-fold: First, 2 out of the 3 non-Russian 
crewmembers were slated to go home on May 13th on Soyuz 33 S, leaving the ISS with no USOS-based EVA 
capability until Soyuz 35S docking in late May. Second, there was hope that if the crew was able to access the leak 
site within the short time frame, the NH 3 would remain a single-phase liquid for long enough to allow the crew to 
see NH 3 flakes and determine a possible leak source. In the end this hope was dashed as even though the NH 3 
remained single-phase, the visible “snow” stopped within hours of 2B PVTCS shutdown. Other factors contributing 
to the decision to conduct an EVA on May 1 1 th were the facts that the EVA was a fairly straightforward changeout 
of a single ORU with the spare PFCS located conveniently nearby, and the crewmembers were familiar with the P6 
truss site from past experience. 
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B. US EVA 21 - May 11 th , 2013 

Upon arriving at the P6 truss worksite, the crew found no evidence of NH 3 leakage - all ORUs and structures 
were reported to be “really, really clean” by EVA crewmember Chris Cassidy. Approximately 3 hours after airlock 
hatch open - and less than 49 hours from the time when the crew first notified MCC-H of a visible leak - the new 
pump was installed and powered on. As the spacewalk was running ahead of schedule, MOD decided, with 
engineering concurrence, that the pump would be activated while the crew was near the worksite. After pump 
activation, the crew stayed in position near the worksite for approximately 30 minutes looking for signs of a NH 3 
leak, and saw nothing. 

The installation of the new PFCS resulted in a slight increase in the total system fluid mass thanks to the stored 
NH 3 within that unit. Overall, however, system quantities were several percent lower than before the May 9th leak 
started. 

From an operational standpoint, the 
PFCS removal & replacement resulted in a 
negligible amount of post-replacement 
retraining and procedure updates. The only 
additional operations work required was that 
new calibration data needed to be uploaded 
to software which communicated with the 
PFCS to allow it to properly read sensor data 
utilized in onboard FDIR algorithms. Until 
that calibration data was uplinked, operations 
personnel inhibited this software-based 
PFCS FDIR which protected against 
overtemperature and undertemperature 
exceedances. MOD & engineering personnel 
were comfortable with inhibiting these FDIR 
algorithms for the days it took the new 
calibrations to be uplinked, for a few reasons. 

First, as noted previously, PVTCS 2B was a 
warm-biased loop and undertemperature was 
seen as an extremely remote possibility. 

Overtemperature exceedance was possible, 
but if it would occur it would take several 
hours. Finally, the 2B channel was not yet 
providing electrical power to its normal 
downstream loads and therefore did not 
require quick response in the event of a 
pump failure in order to maintain critical avionics. MOD and engineering personnel thus decided that the benefits of 
providing 2B channel ORU cooling while awaiting calibration updates outweighed the risk of operating without 
software-based fault detection capability. 

VII. Leak Resolution & Present System Status 

Since the replacement of the 2B PFCS in May of 2013, 2B PVTCS system mass has stopped decreasing in any 
discemable fashion. Conveniently, the suspect PFCS was stowed in a position which provided data connectivity to 
its sensors, allowing ground personnel to observe as the ORU leaked out its remaining NH 3 content over the next 
several weeks, as shown in Figure 8. 

There are no plans to return the leaked PFCS to Earth for a formal failure investigation to determine the root 
cause of the leak. Therefore, this PFCS is destined to remain onboard IS S as a failed ORU. 

As of the writing of this document, the ISS Program has asked MOD and engineering personnel to create a plan 
to retract the TTCR and un-isolate the P6 PVR, as well as possibly performing another refill of the 2B PVTCS, in 
mid-2014. Doing so would reduce the risk to the TTCR, in its role as spare PVR, from micrometeoroid orbital 
debris. Such an EVA would effectively return the 2B PVTCS to a nominal configuration. 


1 - EETCS_LoopB_PF'S Accu» Ran Qtyl 



Figure 8. PFCS S/N 4 telemetry after removal from 2B PVTCS 
(GMT 2013/123-163). As the failed PFCS was transferred to a 
stowage site with data connectivity, MCC-H was able to monitor as 
the pump continued to lose ammonia. Point A shows the quantity 
level (%) and pressure (kPa) at the time of install during EVA 21. At 
point B, the PFCS accumulator bottoms out and the ammonia 
becomes a two-phase liquid-gas, with pressure dependent on thermal 
environment. Leakage continued until approximately GMT 180 when 
pressure dropped to vacuum. 
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VIII. Conclusion 


From the start of the small leak as early as 2004, to the refill in 201 1, to the drastic system architecture change of 
US EVA 20 in 2012, and finally the isolation and termination of the leak in May of 2013, it is remarkable to note 
that the 2B PVTCS remained active for all but the equivalent of a few days: during EVA 20, and during the large 
leak immediately before EVA 21. The forethought which went into the design of an in-situ PVTCS refill pathway, 
as well as the capabilities of the EAS and its jumpers, is impressive. 

It must be noted, however, that the use of ground-commandable motorized valves in place of EVA-operated 
valves (and the EAS jumpers) could have theoretically prevented all but one of the EVAs executed to service the 2B 
PVTCS. When the leak was discovered in 2008, if the relevant valves had been ground-commandable, MCC-H 
could have isolated the 2B lines of the P6 PVR, cross-connected 2B to the EETCS, and monitored for years to see if 
the radiator or the PVTCS system was leaking. Eventually, the PFCS would have been deemed the likely culprit 
and would have been replaced via EVA. This admittedly is an operations-centric perspective and does not take into 
account the increased risk of leakage due to the valves nor any design challenges which would have been involved. 

The overall lesson is in the importance of bringing the operations mindset into the design process as early in the 
project life cycle as possible to assist in evaluating the critical failure points not just in the final product, but in the 
assembly sequence as well. This lesson was on display for ISS by the design engineers who provided the PVTCS 
refill-from ETCS capability, and later by the MOD personnel who devised clever failure mitigations even after 
hardware was being constructed. The design process benefits from having a stakeholder with a different perspective 
to help create the final product and to understand its capabilities. In retrospect, this combination of versatile design 
and clever mitigation strategies have allowed the PVTCS to provide the reliability demanded by the ISS Program 
despite suffering an ongoing and potentially capability-limiting failure. 
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• STS- 134 EVA#2 

■ May 22 nd 201 1 

■ Goal: 

— Top off the 2B FVTCS by adding approximately 
8lbmof NH3 

■ Method: 

— Deliver a regulated ammonia charge from one of 
the central External Thermal Control System 
(ETCS) loops via a series of built-in oonduits and 
EVA-installed jumpers from the PI truss to the P6 
truss 

■ Time to Han: 

— Approximately 20 months 
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• US Stage EVA #20 

■ November 1 st 2012 

■ Goal: 

— Attempt to isolate accelerated FVTCS 2B Leak 

■ Method: 

— Qose valve to disconnect 2B PVTCS from radiator Expedition 33 patch 

— Utilize fluid jumpers to reroute ammonia to an 
adjacent decommissioned ammonia loop with its 
own radiator 

— Deploy that radiator 

■ Time to Han: 

— Approximately 5 weeks 
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• US Stage EVA #21 

■ May 11 th 2013 

■ Goal: 

— Stop 9brrVday visible overboard leak 

■ Method: 



Expedition 35 Patch 


— Remove and replace 2B Pump Row Control 
Subassembly (PFCS) with spare. . . 

— . . .unless EVA crew determined it to be obvious that 
any other part of P6 truss was the culprit 


Time to Han: 

— Less than 48 hours 
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• Leak Status: 

■ The 2B FVTCS leak was isolated to PFCS S/N 0004 

■ Since May 2013, system mass has not appreciably changed 

■ An EVA may occur as early as this summer to retract the 
TTCR and return 2B FVTCS to design configuration 


• Conclusion: 

■ ISS FVTCS leak mitigation capabilities were crucial to 
continuing successful operation 

■ An operations mindset during the design process is a 
necessity 
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